feat(gooddata-eval): generate eval datasets from insights, plus a readable HTML report - #1808

A run's output was a pass/fail table and a turn count, which is enough to
know that something failed and not enough to know why. Reading a failure
meant re-running the item by hand against the live agent.

`gd-eval report`, and `run --html`, write one self-contained HTML file for a
run or several side by side. Per item it shows the whole conversation rather
than a count: every tool call with the arguments it was invoked with, every
reasoning step, and each step's own wall time in execution order, so the
pipeline the agent actually followed is visible. `--redact` drops conversation
and response ids and raw reasoning and renames models to Model A/B for output
that can leave the building.

`timeline_detail` builds the breakdown and the tool calls together from one
event list. They are index-joined -- the timeline carries only a name, and
the join back to arguments is by index -- so building them apart would let
them drift silently.

Two wall-clock caps bound a run. httpx's timeout is per-read, so an agent
streaming reasoning events resets it on every chunk and a runaway item ran
815s under a 300s client timeout. `--turn-timeout` bounds one turn and
`--item-timeout` is a hard ceiling across all of an item's turns, anchored at
conversation creation so a multi-turn item cannot spend N times the budget.
Both default to uncapped, and `TurnTimeoutError` is not retried: a cap that
fires is a verdict, not a transient failure.

jira: AIS-48
risk: low
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Hand-authoring eval questions means writing a question and then guessing the
metric, dimension and filter it should produce, which is how a dataset ends up
full of questions the data model cannot answer.

`gd-eval generate` inverts that. It reads the charts a customer already built
via the declarative analytics model, translates each visible insight's buckets,
sorts and filters into an `expected_output.visualization` spec, and only then
asks an LLM to write the analyst question that chart answers. The expected
output is copied out of a live object rather than invented, so every question
is answerable in the real LDM by construction and the LLM only writes English.

Anything inexpressible is skipped with a printed reason rather than
approximated, and a question that contradicts its own spec is a hard error:
ranking words require a real sort, filter words a real filter, a breakdown
clause a non-empty view_by. One rewrite is attempted, then the item is dropped.

`--enrich-ranked N` derives ranked items, because analysts sort in Analytical
Designer and save without persisting the sort, leaving that coverage near zero
on real models. Adding a limit to a definition that already executes cannot
make it unanswerable. The budget goes to the best-grounded first: insights
whose own title promised a ranking their definition never implemented, then
ranking filters added to a plain breakdown. Derived items carry `derived_from`
and `derived_basis` so a pass rate over them stays separable.

Only a ranking filter is derived, never a sort: on the same base a ranking is
the stronger item, since "the top 3 X by Y" has one correct spec while "X
sorted by Y" leaves the direction to the reader.

The sort a question asks for is now graded, which it previously was not.
`sort_by` was written into every fixture and read by nobody -- the evaluator
loads `expected_output.visualization` into `CreatedVisualization`, which
declared no such field and is configured `extra="ignore"`, so pydantic
discarded it and `strict_pass` covered cross-references, metrics, dimensions,
filters and chart type only. A question saying "sorted by Order id ascending"
-- seven of forty on one real workspace -- asked for something no check saw.

`AacQuery` gains `sort_by`, `check_sorts` compares it, and `strict_pass` counts
it. Entries stay raw dicts for the same reason `filter_by` does: the agent adds
keys this does not read, and a typed model would reject a chart that is
otherwise correct. The comparison uses the shape the agent emits, taken from
recorded runs: `{type: metric_sort, direction, metrics: [alias]}` and `{type:
attribute_sort, direction, by: alias}`, with one build sending both `by` and
`metrics` on a metric sort -- so the entry's own `type` decides which key names
the fields, never whichever key is present. Aliases resolve to uris and date
granularities fold to one spelling, as filters already do, and order is
significant: sorted by region then revenue is not sorted by revenue then
region.

The check is deliberately not symmetric with the filter ones. An empty
`sort_by` records that the fixture has no sort, not that the chart must be
unsorted -- a generated item inherits that emptiness from an insight whose
author sorted in Analytical Designer and saved without the sort sticking. A
spurious filter changes which rows a reader sees and is always wrong, while a
volunteered sort changes only their order, and ascending on a time axis is what
any renderer picks unprompted. So a required sort is enforced and a volunteered
one is free; the cost is that a wrong sort over an unsorted fixture goes
ungraded, the lesser error while `[]` cannot distinguish "unsorted" from
"unrecorded".

A tiebreak the agent appends after the recorded sorts is free as well: "state
descending" is satisfied by "state descending, then city", so the recorded
sorts must lead and match in order, and anything after them is not compared.

Grading a previously ungraded dimension means items that passed while omitting
a sort their fixture records now fail. Re-baseline before comparing a run
against an older one.

The declarative-to-AAC mapping is not ours. `convert()` calls the platform's
own `declarative_visualization_to_aac()` (gooddata-code-convertors, via
gooddata-sdk): both definitions the evaluator compares are platform output, so
the platform's conversion is the right owner, and on one production workspace
it converts every insight where the hand-written mapping had covered 44 of 62.
What stays ours is deciding what the evaluator cannot yet score -- derived
measures, measure-level filters, a repeater's label among its metrics -- each
skipped with a printed reason, and stripping the no-op filters AD saves for an
"All" selection, which would otherwise let a question claim a filter its chart
lacks. A map's `location` bucket is skipped on purpose: it holds a rendering
label, and a question built from it reads as "broken down by City pushpin
latitude". Chart type names are the convertor's, which are also the agent's.
One granularity is patched: the convertor maps `GDC.time.week_us` to `WEEK_US`,
which is not a platform enum; the SDK's own table says `WEEK`.

Two classes of question are unwinnable however well the agent behaves, and both
are reported: a name the model carries more than once (one workspace has six
labels titled "Product Title"), droppable with `--skip-ambiguous`; and a date
granularity's cyclical twin, since MONTH walks consecutive months while
MONTH_OF_YEAR stacks every January. Granularities move to `core/granularity.py`,
shared with scoring, which also folds `attribute/x.month` and `label/x.month`
to one uri -- a date dataset exposes each granularity as an attribute whose
only label carries the same id, and comparing the raw strings failed a chart
that was correct.

Alias resolution and the uri-to-title fallback come from `core/scoring.py`:
one copy, which is the one the evaluator scores against.

The package's `AGENTS.md` gains a section running the whole pipeline --
generate, run, report, models -- with the connection precedence, the snapshot
loop that makes generation iterable offline, and the environment variables each
subcommand reads. `generate` and `report` were not mentioned there at all.

`openai` joins the package's `dev` dependency group, so a plain `uv run` in a
fresh clone has the phrasing step and the LLM judge without naming the extra.
It stays under `optional-dependencies` for anyone installing from PyPI, and
every import site is still guarded or deferred.

Three fixes from the first live run over a customer workspace. A ranking filter
with no `attribute` ranks the full dimension tuple, so the "top N <dimension>"
shorthand in the writer's brief is only true with one dimension; with two it
told the writer a within-group scope the filter lacks, and the agent built what
the question said. `CreatedVisualization.id` is optional: the agent sometimes
omits it, nothing scores on it, and a required field turned a scorable chart
into an errored item.

jira: AIS-48
risk: high
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hbj7SaGm6ucimov4NeMwqt